System and method for a high performance color filter mosaic array

ABSTRACT

A method of implementing high-performance color filter mosaic arrays (CFA) using luminance pixels. The introduction of luminance pixels greatly improves the accuracy of the image acquisition process for a given pixel and image sensor size.

RELATED APPLICATIONS

This application claims priority under 35 U.S.C. §1.119(e) toprovisional application 60/725,334, filed Oct. 13, 2005 by SorinDavidovici, and further to application Ser. No. 11/549,199 filed Oct.13, 2006, both incorporated herein by reference.

FIELD OF THE INVENTION

This invention relates generally to the field of photometry and moreparticularly to a high-performance color filter mosaic arrays (CFA).

BACKGROUND OF THE INVENTION

Photometry deals with the measurement of visible light. The human eyecan only see light in the visible spectrum and has differentsensitivities to light of different wavelengths within the spectrum.When adapted for bright conditions (photopic vision), the eye is mostsensitive to greenish-yellow light at 555 nm.

The human eye has three types of color receptor cells (cones) thatrespond to incident radiation with different spectral response curves. Afourth type of cell (rod) is also present but it plays no role in colorvision. The existence of exactly three types of color receptor cellsimplies that three numerical components are necessary and sufficient todescribe a perceived color.

Digital images are comprised of a number of picture elements or pixels.Pixel values in color images are specified by tristimulus values.Tristimulus values are the amounts of three primaries that specify aperceived color. The colors red, green and blue form a preferred set ofthree primary colors.

A pixel is, generally, the smallest addressable unit of an image on asolid state imager, display screen or in a bitmapped image. They arerated by their number of horizontal and vertical pixels; for example,1024×768 means 1024 pixels are displayed in each row, and there are 768rows (lines). Many image-acquisition and display systems are not capableof displaying the different color channels at the same site. Thisapproach is generally resolved by using multiple sub-pixels, each ofwhich handles a single color channel. With color systems, each pixelusually contains values for at least three, e.g., red, green and blue,subpixels.

It is desirable to reduce the average number of physical subpixelscontained within one image sensor pixel, and hence image sensor siliconarea, while not significantly reducing image quality. The mostwidespread method to give red, green and blue color sensitivity to imagesensors is the application of a color filter mosaic array (CFA) on topof an image sensor. The CFA comprises an array of filters. Each filterlimits the wavelengths of light that is provided to their associatedimage sensor. The most common implementation is the three color red,green, blue (RGB) pattern. Other color implementations exist such asthree-color, e.g., yellow, magenta, cyan, complementary patterns ormixed primary/complementary colors and four color systems where thefourth color is a white or a color with shifted spectral sensitivity.

Although many criteria of a technical and physical implementation naturecould be applied to choose a CFA pattern some of the most important areimmunity to color artifacts and moiré patterns, minimization of patterninteraction with image sensor imperfections, color reconstructioncomputational complexity and immunity to optical and electrical crosstalk between neighboring pixels. Preferred candidate color patternsshould therefore have all three, red, green and blue, componentsavailable in the neighborhood of each pixel, each pixel should have thesame number of neighbors of a given color and diagonal pixel alignmentis desirable whenever possible.

FIG. 1 illustrates one of the most common CFA patterns in use today, theBayer pattern. Each pixel is covered with an individual red, green orblue filter. Thus each image sensor, or pixel, captures only one color;full color values for each pixel are determined by interpolation usingsurrounding pixel values. Pixels 110 (red), 120 and 130 (green) and 140(blue) form a basic Bayer pattern that is repeated multiple times toinstantiate a practical image sensor. Compared to a monochrome sensorwith the same pixel count and dimensions, the CFA approach lowers theavailable spatial resolution by roughly 30% to 40% and it requiresinterpolation calculations to reconstruct the color values for eachpixel.

While the RGB color model is sufficient for computer graphics renderingof images, it is widely recognized that the YUV color model moreaccurately models the human perception of color. The YUV color modeldefines a color space in terms of one luminance (Y) and two chrominance(UV) components (saturation and hue). Digital cameras typically convertthe RGB pixel values into YUV components using a picture reconstructionprocess that approximates the luminance (Y) channel by the green (G)pixels. The luminance for the non-G pixels is approximated by simpleinterpolation. The chrominance of the red and blue pixels is calculatedas Cr=R−Y and Cb=B−Y using the approximate luminance values. The Cr andCb values are spatially filtered and the missing values are obtained byinterpolation. At this time all pixel should have their luminance andchrominance values and their RGB values are computed by R=Y+Cr, B=Y+Cband G from Y=(R+G+B)/3 or from Y=0.2R+0.7G+0.1B or from some othersimilar formula. It is apparent from the above that the Bayer CFAapproach trades off accuracy and resolution for simplicity.

Although many variants exist on the above method of deriving individualpixel color they all suffer from the fundamental limitation of the BayerCFA that luminance is not directly available, but rather must berecreated from dispersed green, red and blue chroma information. Becausethe human eye is most sensitive to luminance information, the imagesprovided using prior art methods are sub-optimal.

SUMMARY OF THE INVENTION

According to one aspect of the invention, a pixel includes a luminancefilter comprising a plurality of colors, wherein the size of each of thecolors of the luminance filter is proportional to a share of the colorused to calculate luminance using the plurality of colors.

According to another aspect of the invention, a color filter arrayincludes a luminance pixel, the luminance pixel comprising a luminancefilter including a plurality of colors, wherein the size of each of thecolors of the luminance filter is proportional to a share of the colorused when calculating luminance using the plurality of colors.

With such an arrangement, luminance information can be directly measuredat each pixel, thereby providing a resultant output image with highresolution and accuracy. These and other advantages of the presentinvention will be described in more detail below with regard to theattached Figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a diagram of a Bayer pattern Color Filter Array of the priorart;

FIG. 2 is a diagram illustrating several embodiments of a luminance (Y)filter of the present invention;

FIG. 3 is a high level block diagram illustrating components of aluminance pixel of the present invention;

FIG. 4 is a diagram of one embodiment of a Color Filter Array using theluminance filter of FIG. 2;

FIG. 5 is a diagram of a second embodiment of a Color Filter Array ofthe present invention that uses the luminance filter of FIG. 2.

DETAILED DESCRIPTION

According to one aspect of the invention, a luminance filter is providedwhich will enable luminance pixel information to be directly obtainedfor an image using a Color Filter Array (CFA). The introduction ofluminance pixels greatly improves the accuracy of the image acquisitionprocess for a given pixel and image sensor size. Using the luminancefilters of the present invention, luminance can be accurately capturedat high resolution with high accuracy. Further objects and advantages ofmy invention will become apparent from a consideration of the drawingsand ensuing description.

One aspect of the invention is the realization that even though thehuman eye is more sensitive to pixel luminance information then to pixelchrominance information, this fact which is not exploited by the BayerCFA. An improved CFA design of the present invention emphasizesluminance information acquisition over chrominance informationacquisition—greatly improving the overall perceptual picture quality andreducing the sensor silicon area requirement for a given imageacquisition resolution requirement. A luminance (Y) pixel mask, providedin any one of a number of varieties, may be used to directly acquireluminance information.

The luminance pixel includes a luminance (Y) mask which is positionedover an image sensor to allow the image sensor to directly extractluminance data from received light wavelengths. The present inventiontherefore removes complexity and inaccuracies inherent in theinterpolated process of the prior art, while providing high resolutionoutput.

FIGS. 2A-2C illustrate several optical filter mask patterns for the Ypixel. Although the Y pixel masks illustrated in FIGS. 2A-C utilize red,green and blue components, these components are provided by way ofexample only. Other color combinations, such as three-color, e.g.,yellow, magenta, cyan, complementary patterns or mixed primary andcomplementary colors or four color systems where the fourth color is awhite or a color with shifted spectral sensitivity or other colorcombinations could be implemented as well, and the present invention isnot limited to any particular color combination.

The optical filter mask may be formed from a sheet of dyed glass,gelatin, plastic or other material used to absorb certain colors andpermit better rendition of others. The specific shape and pattern of theoptical filter masks illustrated in FIGS. 2A-2C are not significant aslong as the resultant proportions of the filtered colors are maintained.The Y pixel definition assumed here is Y=0.2R+0.7G+0.1B. Thus the redarea of the filter is twice as large as the blue area of the filter andthe green area of the filter is seven times as large as the blue area ofthe filter. Other ratios of component colors could be used as well,according, among other factors, to the assumed human visioncharacteristics.

The shape of the pixels illustrated in FIGS. 2A-2C is arbitrary. FIG. 2Aillustrates a rectangular pixel, FIG. 2B a square and FIG. 2C a circularpixel shape but other shapes which may be dictated by particular designsare included within the scope of this invention as equivalents.

FIG. 3 is a diagram provided to illustrate components of a Y pixel ofthe present invention. The Y pixel 40 includes a Y pixel mask 42 and animage sensor 44. The Y pixel mask is a mask which includes a colorfilter having multiple colors, with the size of each one of the multiplecolors of the mask being selected in accordance with a proportion of thecolor wavelength that is included in a luminance calculation. The imagesensor may be implemented using a CCD (charge-coupled device) or CMOS(complementary metal oxide semiconductor) technology. In a preferredembodiment, the image sensor is a High Dynamic Range Sensitive mageSensor such as that disclosed in patent application Ser. No. 11/533,866or a High Dynamic Range Sensitive Sensor with Gain Control such as thatdisclosed in patent Ser. No. 11/533,870, both filed Sep. 21, 2006 byDavidovici, and incorporated herein by reference. As shown in FIG. 3, aslight impinges on the pixel mask, the photons associated with the imagewill be provided to the image sensor in proportion to the luminancevalue of the image.

The luminance pixels of FIGS. 2A-2C, or their equivalents may bearranged in a variety of patterns with a variety of other types ofpixels in a Color Filter Array. One preferred embodiment of a luminance(or Y) based CFA pattern is illustrated in FIG. 4. Pixels 210, 230, 250,270, 280, 290, 310, 320, 330, 340, 250 and 360 (Y, or luminance pixels),220, 240 (red pixels) and 260, 300 (blue pixels) form the basic patternthat is repeated multiple times to instantiate a practical image sensor.This preferred CFA pattern exploits the psychovisual properties of thehuman eye and eliminates drawbacks associated the Bayer CFA pattern.

The introduction of the Y pixels also serves to directly increase theimage resolution. As evident in FIG. 4 each chroma pixel, i.e. red (R)or blue (B), is surrounded by luminance (Y) pixels. The availability ofluminance information from all spatial directions ensures accuraterecovery of luminance information at the chroma pixel locations. By wayof example, luminance values at chroma pixel location can be derivedusing well known methods based on interpolation or mathematicaltechniques.

As evident in FIG. 4 each luminance pixel is bordered by either two orfour chroma pixels. The proximity of chroma information to the luminancepixels ensures accurate recovery of chrominance information at theluminance pixel locations using well known methods based oninterpolation or other mathematical techniques. The presence of chromainformation within the luminance pixel value will further aid theaccurate recovery of chrominance information at luminance pixellocations.

Yet another preferred CFA implementations utilizing the disclosedluminance pixels is illustrated in FIG. 5. Pixels 500 (red pixel), 540(blue pixel) and 520, 530 (Y pixels) form the basic pattern that isrepeated multiple times to instantiate a practical image sensor. Thispreferred CFA pattern also exploits the psychovisual properties of thehuman eye and eliminated drawbacks associated the Bayer CFA pattern. Thenew Y pixels which are composed of a mixture of green and red and blueinformation are introduced. Y pixels directly contain the luminanceinformation to which the human eye is most sensitive thereforeincreasing the effective resolution of the image sensor.

As evident in FIG. 5 each luminance pixel is bordered by four chromapixels. The proximity of chroma information to the luminance pixelsensures accurate recovery of chrominance information at the luminancepixel locations using well known methods based on interpolation or othermathematical techniques. The presence of chroma information within theluminance pixel value will further aid the accurate recovery ofchrominance information at luminance pixel locations.

Accordingly, the present invention is related to a method ofimplementing CFA filters that greatly increases image resolution byusing luminance pixels. Other CFA implementations of a similar nature,in addition to those of FIGS. 3 and 4 will be obvious to one skilled inthe art.

Having described various embodiments of the invention, it will beappreciated that although certain components and process steps have beendescribed the descriptions are representative only; other functionaldelineations or additional steps and components can be added by one ofskill in the art, and thus the present invention should not be limitedto the specific embodiments disclosed. The various representationalelements may be implemented in hardware, software running on a computer,or a combination thereof and modification to and variation of theillustrated embodiments may be made without departing from the inventiveconcepts herein disclosed. Accordingly, the invention should not beviewed as limited except by the scope and spirit of the appended claims.

The invention claimed is:
 1. A pixel array comprising: a pixel; a singlepixel luminance filter positioned over the pixel, the single pixelluminance filter comprising a plurality of colors, wherein the size ofeach of the colors of the single pixel luminance filter is proportionalto a share of the associated color of the single pixel luminance filterthat is used to calculate a luminance value for the pixel, and whereinthe luminance value for the pixel is determined using only filteredelectromagnetic radiation received through the single pixel luminancefilter.
 2. The pixel of claim 1 including an image sensor, coupled toreceive a luminance filtered electromagnetic radiation signal from thesingle pixel luminance filter, and to provide an output indicative of aluminance of an input electromagnetic radiation signal.
 3. The pixel ofclaim 1, wherein the plurality of colors include red, green and blue,and wherein the green filter comprises 70% of the single pixel luminancefilter, the red filter comprises 20% of the single pixel luminancefilter and the blue filter comprises 10% of the single pixel luminancefilter.
 4. A color filter array comprising: a luminance pixel, theluminance pixel comprising a single pixel luminance filter including aplurality of colors, wherein the size of each of the colors of thesingle pixel luminance filter is proportional to a share of the colorused when calculating a luminance value of the luminance pixel, andwherein the luminance value is determined using only filteredelectromagnetic radiation received through the single pixel luminancefilter.
 5. The color filter array of claim 4 wherein the luminance pixelincludes an image sensor, coupled to receive a luminance filteredelectromagnetic radiation signal from the single pixel luminance filter,and to provide an output indicative of a luminance of an inputelectromagnetic radiation signal.
 6. The color filter array of claim 4,wherein the plurality of colors include red, green and blue, and whereinthe green filter comprises 70% of the single pixel luminance filter, thered filter comprises 20% of the single pixel luminance filter and theblue filter comprises 10% of the single pixel luminance filter.
 7. Thecolor filter array of claim 4, further comprising: a plurality ofluminance pixels; and a plurality of chroma pixels, wherein each chromapixel is adjacent to the at least two luminance pixels.
 8. The colorfilter array of claim 7, wherein each chroma pixel is adjacent to atleast four luminance pixels.
 9. The color filter array of claim 8,wherein the luminance pixels surround each of the plurality of chromapixels.